{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "colab": {
   "name": "quickstart_Pendulum-v1.ipynb",
   "provenance": [],
   "collapsed_sections": [],
   "include_colab_link": true
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "accelerator": "GPU"
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "view-in-github",
    "colab_type": "text"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/AI4Finance-Foundation/ElegantRL/blob/master/quickstart_Pendulum_v1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c1gUG3OCJ5GS"
   },
   "source": [
    "# **Pendulum-v1 Example in ElegantRL-HelloWorld**\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DbamGVHC3AeW"
   },
   "source": [
    "# **Part 1: Install ElegantRL**"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "U35bhkUqOqbS",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "e509246b-ab75-40f3-9484-9ff67f115238"
   },
   "source": [
    "# install elegantrl library\n",
    "!pip install git+https://github.com/AI4Finance-LLC/ElegantRL.git"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Collecting git+https://github.com/AI4Finance-LLC/ElegantRL.git\n",
      "  Cloning https://github.com/AI4Finance-LLC/ElegantRL.git to /tmp/pip-req-build-2yt51wrh\n",
      "  Running command git clone -q https://github.com/AI4Finance-LLC/ElegantRL.git /tmp/pip-req-build-2yt51wrh\n",
      "Requirement already satisfied: gym in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (0.17.3)\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (3.2.2)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (1.19.5)\n",
      "Collecting pybullet\n",
      "  Downloading pybullet-3.2.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (90.8 MB)\n",
      "\u001b[K     |████████████████████████████████| 90.8 MB 258 bytes/s \n",
      "\u001b[?25hRequirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (1.10.0+cu111)\n",
      "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (4.1.2.30)\n",
      "Collecting box2d-py\n",
      "  Downloading box2d_py-2.3.8-cp37-cp37m-manylinux1_x86_64.whl (448 kB)\n",
      "\u001b[K     |████████████████████████████████| 448 kB 61.2 MB/s \n",
      "\u001b[?25hRequirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym->elegantrl==0.3.3) (1.3.0)\n",
      "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym->elegantrl==0.3.3) (1.5.0)\n",
      "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from gym->elegantrl==0.3.3) (1.4.1)\n",
      "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym->elegantrl==0.3.3) (0.16.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (0.11.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (1.3.2)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (2.8.2)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (3.0.6)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib->elegantrl==0.3.3) (1.15.0)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch->elegantrl==0.3.3) (3.10.0.2)\n",
      "Building wheels for collected packages: elegantrl\n",
      "  Building wheel for elegantrl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for elegantrl: filename=elegantrl-0.3.3-py3-none-any.whl size=181145 sha256=9b2b00173e3b6342690be9edb79f6e19703182f2782ee595755e13bbd6c18ef0\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-05zy5chw/wheels/52/9a/b3/08c8a0b5be22a65da0132538c05e7e961b1253c90d6845e0c6\n",
      "Successfully built elegantrl\n",
      "Installing collected packages: pybullet, box2d-py, elegantrl\n",
      "Successfully installed box2d-py-2.3.8 elegantrl-0.3.3 pybullet-3.2.1\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UVdmpnK_3Zcn"
   },
   "source": [
    "# **Part 2: Import Packages**\n"
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "from elegantrl_helloworld.demo import *\n",
    "\n",
    "gym.logger.set_level(40)  # Block warning"
   ],
   "metadata": {
    "id": "AAPdjovQrTpE"
   },
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3n8zcgcn14uq"
   },
   "source": [
    "# **Part 3: Specify Agent and Environment**\n",
    "\n",
    "*   **agent**: chooses a agent (DRL algorithm) from a set of agents in the [directory](https://github.com/AI4Finance-Foundation/ElegantRL/tree/master/elegantrl/agents).\n",
    "*   **env**: creates an environment for your agent.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "E03f6cTeajK4"
   },
   "source": [
    "env = PendulumEnv(\"Pendulum-v0\", target_return=-500)\n",
    "args = Arguments(AgentSAC, env)"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "# **Part 4: Specify hyper-parameters**\n",
    "A list of hyper-parameters is available [here](https://elegantrl.readthedocs.io/en/latest/api/config.html)."
   ],
   "metadata": {
    "id": "rcFcUkwfzHLE"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "args.reward_scale = 2**-1  # RewardRange: -1800 < -200 < -50 < 0\n",
    "args.gamma = 0.97\n",
    "args.target_step = args.max_step * 2\n",
    "args.eval_times = 2**3"
   ],
   "metadata": {
    "id": "9WCAcmIfzGyE"
   },
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z1j5kLHF2dhJ"
   },
   "source": [
    "# **Part 5: Train and Evaluate the Agent**\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "KGOPSD6da23k",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "1c9cc4fe-bbc8-4dc4-aa37-346ab110c5ab"
   },
   "source": [
    "train_and_evaluate(args)"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "| Arguments Remove cwd: ./Pendulum-v0_SAC_0\n",
      "################################################################################\n",
      "ID     Step    maxR |    avgR   stdR   avgS  stdS |    expR   objC   etc.\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/elegantrl_helloworld/run.py:309: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)\n",
      "  s_tensor = torch.as_tensor((state,), dtype=torch.float32, device=device)\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "0  4.00e+02-1231.15 |\n",
      "0  4.00e+02-1231.15 |-1231.15  265.6    200     0 |   -2.75   9.18   0.56   1.00\n",
      "0  4.52e+04-1043.38 |\n",
      "0  4.52e+04-1043.38 |-1043.38  100.8    200     0 |   -2.10  48.81 -51.44   0.74\n",
      "0  6.64e+04 -512.52 |\n",
      "0  6.64e+04 -512.52 | -512.52  122.9    200     0 |   -1.40  12.07 -66.04   0.54\n",
      "0  8.28e+04 -197.82 |\n",
      "0  8.28e+04 -197.82 | -197.82  102.3    200     0 |   -0.94  21.87 -64.79   0.55\n",
      "| UsedTime: 387 | SavedDir: ./Pendulum-v0_SAC_0\n",
      "| ReplayBuffer save in: ./Pendulum-v0_SAC_0/replay_0.npz\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JPXOxLSqh5cP"
   },
   "source": [
    "Understanding the above results::\n",
    "*   **Step**: the total training steps.\n",
    "*  **MaxR**: the maximum reward.\n",
    "*   **avgR**: the average of the rewards.\n",
    "*   **stdR**: the standard deviation of the rewards.\n",
    "*   **objA**: the objective function value of Actor Network (Policy Network).\n",
    "*   **objC**: the objective function value (Q-value)  of Critic Network (Value Network)."
   ]
  }
 ]
}